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Your background 


e It is not Kafka 101, so you have some basic Kafka experience 
e Golang experience is not necessary 


AGENDA 


e Basics: partitions, consumers, consumer groups 
* Rebalancing 

e Manual commit 

° Producing in batches 
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Limitation 
e Within the partition the order is guaranteed 


° Between the partitions the order is unpredictable 


Consumer groups 


Several consumers can read 1 topic 
independently 


consumer group 
3 


partition 
1 


consumer group consumer group 
1 2 


Low-leveland high-level consumption 
Low level consumption (without consumer group) 
kcat -C -b localhost:9092 -t test -p О-о beginning 
-b localhost:9092 -t rebalancing-demo -p 0 -o beginning -e 


5 Reached end of topic rebalancing-demo [0] at offset 6: exiting 


-b localhost:9092 -t rebalancing-demo -p 0 -o beginning -e 


% Reached end of topic rebalancing-demo [0] at offset 6: exiting 


Low-leveland high-levelconsumption 


High-level consumption (with consumer group) 
kcat -G test-group -b localhost:9092 test 


kcat -G rebalancing -b localhost:9092 -e rebalancing-demo 


% Waiting for group rebalance 
% Group rebalancing rebalanced (memberid rdkafka-b32f75c2-c700-4 
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Reached end of topic rebalancing-demo [0] at offset 8 
% Reached end of topic rebalancing-demo [2] at offset 0 


% Reached end of topic rebalancing-demo [1] at offset 0: exiting 
% Group rebalancing rebalanced (memberid rdkafka-b32f75c2-c700-4 


Last offset always stored for the combination of: 


e Topic 
* Consumer group 
e Partition 


Rebalancing 


Example of partition assigning (1 consumer) 


GROUP TOPIC PARTITION CURRENT-OFFSET CONSUMER-ID 
rebalancing rebalancing-demo 0 rdkafka-dee7f 


rebalancing rebalancing-demo 1 rdkafka-dee7f 
rebalancing rebalancing-demo 2 rdkafka-dee7f 


Example of partition assigning (2 consumers) 


GROUP TOPIC PARTITION CURRENT-OFFSET  CONSUMER-ID 
rebalancing rebalancing-demo © rdkafka-dee7f 


rebalancing rebalancing-demo 1 rdkafka-dee7f 
rebalancing rebalancing-demo 2 sarama-a0104d2 


Rebalancing 


WHEN: EVENTS: 


* Redeployment 
• New consumer is added ° Scaling up(down) 


e Old consumer is gone ° App got stuck 
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Rebalancing 
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Parameters: how Kafka can detect that you app's gone 
° session.timeout.ms 


e kafka heartbeat.interval 


° max.poll.interval.ms 


° sroup.initial.rebalance.delay.ms 


Rebalancing 


heartbeat.interval.ms 


consumer <> 
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Consumer group status 


As usual 


GROUP TOPIC PARTITION CURRENT-OFFSET CONSUMER-ID 
rebalancing rebalancing-demo 0 rdkafka-dee7f 


rebalancing rebalancing-demo 1 rdkafka-dee7f 


rebalancing rebalancing-demo 2 rdkafka-dee7f 


When rebalancing 


Warning: Consumer group 'rebalancing' 15 rebalancing. 
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Rebalancing 


Consume actions (Shopify/sarama) 


func (m Message) ConsumeClaim(session sarama.ConsumerGrour 


for { 

select { 

// 1. Read 

case message, ok := <-claim.Messages(): 
if lok : nil 7 
// 2. Handle 
fmt.Printf( format "topic: %q", message.Topic) 
// 3. Commit 
session.MarkMessage(message, metadata: "") 
session.Commit() 

} 
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Rebalancing 


What should you do after rebalancing event is fired? 


1. Stop reading messages 
2. Consider stop handling 
3. Commit messages (that are already handled) 


Where? 
ConsumeGroupHandler.Cleanup (Shopify/sarama) 


Manual Commit 


Auto-commit 


Shopify/sarama 


config.Consumer.Offsets.AutoCommit.Enable = 
config.Consumer.Offsets.AutoCommit.Interval 


Why you don't like it? 


* Commit regardless of handling status 


true 
= 5 x time.Second 
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Manual Commit 


Concurrent handling 


° 1 goroutine per 1 partition 


Manual Commit 


Can we increase the degree of parallelism? 
e channelBufferSize 


| partition | 


— | OEI 3 


Manualcommit 


What is the correct order of handling? 


* Unsorted » Sorted 


° No way to commit properly + Commit every n-th message 


Manual commit 


What is the correct order of handling? 


° Unsorted | | e Sorted | x ° Sorted w/o gaps 
e No way to commit: '* Commit every n-th 
properly ЕЕ 
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Committed messages 


e You cannot mark only few messages as committed 


ZZ DE = MU Же 


e You сап mark only one message. All the previous ones will be 
considered as committed 
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Manual commit 
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Producing in batches 


segmentio/kafka-go 


writer := &kafka.Writer{ 


Async: false, 

Addr: kafka.TCP( address...: "Localhost:9092"), 
Topic: "test", 

Balancer: &kafka.LeastBytes{}, 

WriteTimeout: writeTimeout, 

BatchSize: 200, 


BatchTimeout: 1 * time.Second, 
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Producing in batches 


segmentio/kafka-go 


var messages []kafkago.Message 


writerWriteMessages(ctx, messages...) 


Producing in batches 


w := &kafka.Writer{ 


Addr: kafka.TCP("localhost:9092", "localhost:9093", "localhost:9094"), 
Topic: "topic-A", 
Balancer: &kafka.LeastBytes{}, 

} 

err := w.WriteMessages(context.Background(), 


kafka.Message{ 

Key: []byte("Key-A"), 

Value: []byte("Hello World!"), 
}, 
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Wrapping up... 


e Rebalancing 
e Consider rebalancing as part of your consumption process 
e Configure session.timeout wisely 
* Commit all processed messages during the rebalancing event 


e Manual commit 
e Use manual commit rather than the automatic one 
° Use only the partitions for paralellism 


e Producing in batches 
e Carefully use the defaults 


About the author 


° Denis Filippov 


e https://t.me/filippov sequel 


° These slides: 


http://bit.do/kafka highload 2022 


Leave your feedback! 


You can rate the talk and give 
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